{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(code-advcd-best-practice)=\n",
    "# Tools for Better Coding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This chapter covers the tools that will help you to write better code. This includes practical topics such as debugging code, logging, linting, and the magic of auto-formatting.\n",
    "\n",
    "As ever, you may need to `conda install packagename` or `pip install packagename` on the terminal before being able to use some of the packages that are featured."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Auto-magically improving your code\n",
    "\n",
    "In the previous chapter, we met the idea of *code style*: even for code that runs, *how* you write it matters for readability. (And it goes without saying that you don't want bugs in your code that stop it running at all.) It is possible to catch some errors, to flag style issues, and even to re-format code to comply with a code style automatically. In this section, we'll see how to use tools to perform these functions automatically.\n",
    "\n",
    "### Linting\n",
    "\n",
    "Linters are tools that analyse code for programmatic and stylistic errors, assuming you have declared a style. A linting tool flags any potential errors and deviations from style before you even run it. When you run a linter, you get a report of what line the issue is on and why it has been raised. They are supposedly named after the lint trap in a clothes dryer because of the way they catch small errors that could have big effects.\n",
    "\n",
    "Some of the most popular linters in Python are [**flake8**](https://flake8.pycqa.org/en/latest/), [**pycodestyle**](https://pycodestyle.pycqa.org/en/latest/intro.html), and [**pylint**](https://www.pylint.org/).\n",
    "\n",
    "Let's see an example of running a linter. VS Code has direct integration with a range of linters. To get going, use `⇧⌘P` (Mac) and then type 'Python Select Linter'. In the example below, we'll use **flake8** (and **pylance**, another VS Code extension). Let's pretend we have a script, `test.py`, containing\n",
    "\n",
    "```python\n",
    "list_defn = [1,5, 6,\n",
    "7]\n",
    "\n",
    "def this_is_a_func():\n",
    "   print('hello')\n",
    "\n",
    "print(X)\n",
    "\n",
    "import numpy as np\n",
    "```\n",
    "\n",
    "To see the linting report, press <kbd>^</kbd> + <kbd>\\`</kbd> (Mac) or <kbd>ctrl</kbd> + <kbd>`</kbd> (otherwise) and navigate to the 'Problems' tab. We get a whole load of error messages about this script, here are a few:\n",
    "\n",
    "- ⓧ missing whitespace after ',' flake8(E231) 1, 15\n",
    "- ⓧ continuation line under-indented for visual indent flake8(E128) 2, 1\n",
    "- ⓧ expected 2 blank lines, found 1 flake8(E302) 4, 1\n",
    "- ⓧ indentation is not a multiple of 4 flake8(E111) 5, 4\n",
    "- ⓧ undefined name 'X' flake8(F821) 7, 7\n",
    "- ⓧ module level import not at top of file flake8(E402) 9, 1\n",
    "- ⓧ 'numpy as np' imported but unused flake8(F3401) 9, 1\n",
    "- ⚠ \"X\" is not defined Pylance(reportUndefinedVariable) 7, 7\n",
    "- ⚠ no newline at end of file flake8(W292) 78, 338\n",
    "\n",
    "each message is a warning or error that says what the problem is (for example, missing whitespace after ','), what library is reporting it (mostly flake8 here), the name of the rule that has been broken (E231), and the line, row position (1, 15). Very helpfully, we get an undefined name message for variable `X`, this is especially handy because it would cause an error on execution otherwise. The same goes for the indentation message (indentation matters!). You can customise your [linting settings](https://code.visualstudio.com/docs/python/linting) in VS Code too.\n",
    "\n",
    "Although the automatic linting offered by an IDE is very convenient, it's not the only way to use linting tools. You can also run them from the command line. For example, for **flake8**, the command is `flake8 test.py`.\n",
    "\n",
    "### Formatting\n",
    "\n",
    "It's great to find out all the ways in which you are failing with respect to code style from a linter but wouldn't it be *even* better if you could fix those style issues automatically? The answer is clearly yes! This is where formatters come in; they can take valid code and forcibly apply a code style to them. This is really handy in practice for all kinds of reasons.\n",
    "\n",
    "The most popular code formatters in Python are probably: [**yapf**](https://github.com/google/yapf), 'yet another Python formatter', from Google; [**autopep8**](https://github.com/hhatto/autopep8), which applies PEP8 to your code; and [**black**](https://black.readthedocs.io/en/stable/), the 'uncompromising formatter' that is very opinionated (\"any colour, as long as it's black\").\n",
    "\n",
    "There are two ways to use formatters, line-by-line (though **black** doesn't work in this mode) or on an entire script at once. VS Code offers an integration with formatters. To select a formatter in VS Code, bring up the settings using <kbd>⌘</kbd> + <kbd>,</kbd> (Mac) or <kbd>ctrl</kbd> + <kbd>,</kbd> (otherwise) and type 'python formatting provider' and you can choose from autopep8, black, and yapf. \n",
    "\n",
    "If you choose **autopep8** and then open a script you can format a *selection* of code by pressing <kbd>⌘</kbd> + <kbd>k</kbd>, <kbd>⌘</kbd> + <kbd>f</kbd> (Mac) or <kbd>ctrl</kbd> + <kbd>k</kbd>, <kbd>ctrl</kbd> + <kbd>f</kbd> (otherwise). They can also (and only, in the case of **black**) be used from the command line. For instance, to use **black**, the command is `black test.py`, assuming you have it installed.\n",
    "\n",
    "Let's see an example of a poorly styled script and see what happens when we select all lines and use <kbd>ctrl</kbd> + <kbd>k</kbd>, <kbd>ctrl</kbd> + <kbd>f</kbd> to auto format with **autopep8**. The contents of `test.py` before formatting are:\n",
    "\n",
    "```python\n",
    "def very_important_function(y_val,debug = False, keyword_arg=0, another_arg  =2):\n",
    "    X = np.linspace(0,10,5)\n",
    "    return X+  y_val +keyword_arg\n",
    "very_important_function(2)\n",
    "\n",
    "list_defn = [1,\n",
    "2,\n",
    "3,\n",
    "5,\n",
    "6,\n",
    "7]\n",
    "\n",
    "import numpy as np\n",
    "```\n",
    "\n",
    "and, after running the auto-formatting command,\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def very_important_function(y_val, debug=False, keyword_arg=0, another_arg=2):\n",
    "    X = np.linspace(0, 10, 5)\n",
    "    return X + y_val + keyword_arg\n",
    "\n",
    "\n",
    "very_important_function(2)\n",
    "\n",
    "list_defn = [1,\n",
    "             2,\n",
    "             3,\n",
    "             5,\n",
    "             6,\n",
    "             7]\n",
    "```\n",
    "\n",
    "So what did the formatter do? Many things. It moved the import to the top, put two blank lines after the function definition, removed whitespace around keyword arguments, added a new line at the end, and fixed some of the indentation. The different formatters have different strengths and weaknesses; for example, **black** is not so good at putting imports in the right place but excels at splitting up troublesome wide lines. If you need a formatter that deals specifically with module imports, check out [**isort**](https://pycqa.github.io/isort/).\n",
    "\n",
    "Apart from taking the pressure off you to always be thinking about code style, formatters can be useful when working collaboratively too. For some open source packages, maintainers ask that new code or changed code be run through a particular formatter if it is to be incorporated into the main branch. This helps ensure the code style is consistent regardless of who is writing it. Running the code formatter can even be automated to happen every time someone *commits* some code to a shared code repository too, using something called a *pre-commit hook*.\n",
    "\n",
    "There is a package that can run **Black** on Jupyter Notebooks too: [**black-nb**](https://pypi.org/project/black-nb/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debugging code\n",
    "\n",
    "Computers are *very* literal, so literal that unless you're perfectly precise about what you want, they will end up doing something different. When that happens, one of the most difficult issues in programming is to understand *why* the code isn't doing what you expected. When the code doesn't do what we expect, it's called a bug.\n",
    "\n",
    "Bugs could be fundamental issues with the code you're using (in fact, the term originated because a moth causing a problem in an early computer) and, if you find one of these, you should file an issue with the maintainers of the code. However, what's much more likely is that the instructions you gave aren't quite what is needed to produce the outcome that you want. And, in this case, you might need to *debug* the code: to find out which part of it isn't doing what you expect.\n",
    "\n",
    "Even with a small code base, it can be tricky to track down where the bug is: but don't fear, there are tools on hand to help you find where the bug is.\n",
    "\n",
    "### Print statements\n",
    "\n",
    "The simplest, and I'm afraid to say the most common, way to debug code is to plonk `print` statements in the code. Let's take a common example in which we perform some simple array operations, here multiplying an array and then summing it with another array:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def array_operations(in_arr_one, in_arr_two):\n",
    "    out_arr = in_arr_one*1.5\n",
    "    out_arr = out_arr + in_arr_two\n",
    "    return out_arr\n",
    "\n",
    "\n",
    "in_vals_one = np.array([3, 2, 5, 16, '7', 8, 9, 22])\n",
    "in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "\n",
    "result = array_operations(in_vals_one, in_vals_two)\n",
    "result\n",
    "```\n",
    "\n",
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "UFuncTypeError                            Traceback (most recent call last)\n",
    "<ipython-input-1-166160824d19> in <module>\n",
    "     11 in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "     12 \n",
    "---> 13 result = array_operations(in_vals_one, in_vals_two)\n",
    "     14 result\n",
    "\n",
    "<ipython-input-1-166160824d19> in array_operations(in_arr_one, in_arr_two)\n",
    "      3 \n",
    "      4 def array_operations(in_arr_one, in_arr_two):\n",
    "----> 5     out_arr = in_arr_one*1.5\n",
    "      6     out_arr = out_arr + in_arr_two\n",
    "      7     return out_arr\n",
    "\n",
    "UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U32'), dtype('<U32')) -> dtype('<U32')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no! We've got a `UFuncTypeError` here, perhaps not the most illuminating error message we've ever seen. We'd like to know what's going wrong here. The `Traceback` did give us a hint about where the issue occurred though; it happens in the multiplication line of the function we wrote.\n",
    "\n",
    "To debug the error with print statements, we might re-run the code like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def array_operations(in_arr_one, in_arr_two):\n",
    "    print(f'in_arr_one is {in_arr_one}')\n",
    "    out_arr = in_arr_one*1.5\n",
    "    out_arr = out_arr + in_arr_two\n",
    "    return out_arr\n",
    "\n",
    "\n",
    "in_vals_one = np.array([3, 2, 5, 16, '7', 8, 9, 22])\n",
    "in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "\n",
    "result = array_operations(in_vals_one, in_vals_two)\n",
    "result\n",
    "```\n",
    "\n",
    "```\n",
    "in_arr_one is ['3' '2' '5' '16' '7' '8' '9' '22']\n",
    "```\n",
    "\n",
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "UFuncTypeError                            Traceback (most recent call last)\n",
    "<ipython-input-2-6a04719bc0ff> in <module>\n",
    "      9 in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "     10 \n",
    "---> 11 result = array_operations(in_vals_one, in_vals_two)\n",
    "     12 result\n",
    "\n",
    "<ipython-input-2-6a04719bc0ff> in array_operations(in_arr_one, in_arr_two)\n",
    "      1 def array_operations(in_arr_one, in_arr_two):\n",
    "      2     print(f'in_arr_one is {in_arr_one}')\n",
    "----> 3     out_arr = in_arr_one*1.5\n",
    "      4     out_arr = out_arr + in_arr_two\n",
    "      5     return out_arr\n",
    "\n",
    "UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U32'), dtype('<U32')) -> dtype('<U32')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can we tell from the values of `in_arr_one` that are now being printed? Well, they seem to have quote marks around them and what that means is that they're strings, *not* floating point numbers or integers! Multiplying a string by 1.5 doesn't make sense here, so that's our error. If we did this, we might then trace the origin of that array back to find out where it was defined and see that instead of `np.array([3, 2, 5, 16, 7, 8, 9, 22])` being declared, we have `np.array([3, 2, 5, 16, '7', 8, 9, 22])` instead and `numpy` decides to cast the whole array as a string to ensure consistency.\n",
    "\n",
    "Let's fix that problem by turning `'7'` into `7` and run it again:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def array_operations(in_arr_one, in_arr_two):\n",
    "    out_arr = in_arr_one*1.5\n",
    "    out_arr = out_arr + in_arr_two\n",
    "    return out_arr\n",
    "\n",
    "\n",
    "in_vals_one = np.array([3, 2, 5, 16, 7, 8, 9, 22])\n",
    "in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "\n",
    "result = array_operations(in_vals_one, in_vals_two)\n",
    "result\n",
    "```\n",
    "\n",
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "ValueError                                Traceback (most recent call last)\n",
    "<ipython-input-3-ebd3efde9b3e> in <module>\n",
    "      8 in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "      9 \n",
    "---> 10 result = array_operations(in_vals_one, in_vals_two)\n",
    "     11 result\n",
    "\n",
    "<ipython-input-3-ebd3efde9b3e> in array_operations(in_arr_one, in_arr_two)\n",
    "      1 def array_operations(in_arr_one, in_arr_two):\n",
    "      2     out_arr = in_arr_one*1.5\n",
    "----> 3     out_arr = out_arr + in_arr_two\n",
    "      4     return out_arr\n",
    "      5 \n",
    "\n",
    "ValueError: operands could not be broadcast together with shapes (8,) (7,) \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still not working! But we've moved on to a different error now. We can still use a print statement to debug this one, which seems to be related to the shapes of variables passed into the function:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def array_operations(in_arr_one, in_arr_two):\n",
    "    print(f'in_arr_one shape is {in_arr_one.shape}')\n",
    "    out_arr = in_arr_one*1.5\n",
    "    print(f'intermediate out_arr shape is {out_arr.shape}')\n",
    "    print(f'in_arr_two shape is {in_arr_two.shape}')\n",
    "    out_arr = out_arr + in_arr_two\n",
    "    return out_arr\n",
    "\n",
    "\n",
    "in_vals_one = np.array([3, 2, 5, 16, 7, 8, 9, 22])\n",
    "in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "\n",
    "result = array_operations(in_vals_one, in_vals_two)\n",
    "result\n",
    "```\n",
    "\n",
    "```\n",
    "in_arr_one shape is (8,)\n",
    "intermediate out_arr shape is (8,)\n",
    "in_arr_two shape is (7,)\n",
    "```\n",
    "\n",
    "```python\n",
    "---------------------------------------------------------------------------\n",
    "ValueError                                Traceback (most recent call last)\n",
    "<ipython-input-4-4961f476c7eb> in <module>\n",
    "     11 in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "     12 \n",
    "---> 13 result = array_operations(in_vals_one, in_vals_two)\n",
    "     14 result\n",
    "\n",
    "<ipython-input-4-4961f476c7eb> in array_operations(in_arr_one, in_arr_two)\n",
    "      4     print(f'intermediate out_arr shape is {out_arr.shape}')\n",
    "      5     print(f'in_arr_two shape is {in_arr_two.shape}')\n",
    "----> 6     out_arr = out_arr + in_arr_two\n",
    "      7     return out_arr\n",
    "      8 \n",
    "\n",
    "ValueError: operands could not be broadcast together with shapes (8,) (7,) \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The print statement now tells us the shapes of the arrays as we go through the function. We can see that in the line before the `return` statement the two arrays that are being combined using the `+` operator don't have the same shape, so we're effectively adding two vectors from two differently dimensioned vector spaces and, understandably, we are being called out on our nonsense. To fix this problem, we would have to ensure that the input arrays are the same shape (it looks like we may have just missed a value from `in_vals_two`).\n",
    "\n",
    "`print` statements are great for a quick bit of debugging and you are likely to want to use them more frequently than any other debugging tool. However, for complex, nested code debugging, they aren't always very efficient and you will sometimes feel like you are playing battleships in continually refining where they should go until you have pinpointed the actual problem, so they're far from perfect. Fortunately, there are other tools in the debugging toolbox...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Icecream and better print statements\n",
    "\n",
    "Typing `print` statements with arguments that help you debug code can become tedious. There are better ways to work, which we'll come to, but we must also recognise that `print` is used widely in practice. So what if we had a function that was as easier to use as `print` but better geared toward debugging? Well, we do, and it's called [**icecream**](https://github.com/gruns/icecream), and it's available in most major languages, including Python, Dart, Rust, javascript, C++, PHP, Go, Ruby, and Java. \n",
    "\n",
    "Let's take an example from earlier in this chapter, where we used a `print` statement to display the contents of `in_arr_one` in advance of the line that caused an error being run. All we will do now is switch out `print(f'in_arr_one is {in_arr_one}')` for `ic(in_arr_one)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from icecream import ic\n",
    "\n",
    "def array_operations(in_arr_one, in_arr_two):\n",
    "    # Old debug line using `print`\n",
    "    # print(f'in_arr_one is {in_arr_one}')\n",
    "    # new debug line:\n",
    "    ic(in_arr_one)\n",
    "    out_arr = in_arr_one*1.5\n",
    "    out_arr = out_arr + in_arr_two\n",
    "    return out_arr\n",
    "\n",
    "\n",
    "in_vals_one = np.array([3, 2, 5, 16, '7', 8, 9, 22])\n",
    "in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "\n",
    "array_operations(in_vals_one, in_vals_two)\n",
    "```\n",
    "\n",
    "```\n",
    "ic| in_arr_one: array(['3', '2', '5', '16', '7', '8', '9', '22'], dtype='<U21')\n",
    "---------------------------------------------------------------------------\n",
    "UFuncTypeError                            Traceback (most recent call last)\n",
    "<ipython-input-6-9efd5fc1a1fe> in <module>\n",
    "     14 in_vals_two = np.array([4, 7, 3, 23, 6, 8, 0])\n",
    "     15 \n",
    "---> 16 array_operations(in_vals_one, in_vals_two)\n",
    "\n",
    "<ipython-input-6-9efd5fc1a1fe> in array_operations(in_arr_one, in_arr_two)\n",
    "      6     # new debug line:\n",
    "      7     ic(in_arr_one)\n",
    "----> 8     out_arr = in_arr_one*1.5\n",
    "      9     out_arr = out_arr + in_arr_two\n",
    "     10     return out_arr\n",
    "\n",
    "UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U32'), dtype('<U32')) -> dtype('<U32')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we get in terms of debugging output is `ic| in_arr_one: array(['3', '2', '5', '16', '7', '8', '9', '22'], dtype='<U21')`, which is quite similar to before apart from three important differences, all of which are advantages:\n",
    "\n",
    "1. it is easier and quicker to write `ic(in_arr_one)` than `print(f'in_arr_one is {in_arr_one}')`\n",
    "\n",
    "2. **icecream** automatically picks up the name of the variable, `in_arr_one`, and clearly displays its contents\n",
    "\n",
    "3. **icecream** shows us that `in_arr_one` is of `type` array and that it has the `dtype` of `U`, which stands for Unicode (i.e. a string). `<U21` just means that all strings in the array are less than 21 characters long.\n",
    "\n",
    "**icecream** has some other advantages relative to print statements too, for instance it can tell you about which lines were executed in which scripts if you call it without arguments:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def foo():\n",
    "    ic()\n",
    "    print('first')\n",
    "    \n",
    "    if 10 < 20:\n",
    "        ic()\n",
    "        print('second')\n",
    "    else:\n",
    "        ic()\n",
    "        print('Never executed')\n",
    "\n",
    "foo()\n",
    "```\n",
    "\n",
    "```\n",
    "ic| <ipython-input-7-8ced0f8fcf82>:2 in foo() at 00:58:19.962\n",
    "ic| <ipython-input-7-8ced0f8fcf82>:6 in foo() at 00:58:19.979\n",
    "first\n",
    "second\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it can wrap assignments rather than living on its own lines:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def half(i):\n",
    "    return ic(i) / 2\n",
    "\n",
    "a = 6\n",
    "b = ic(half(a))\n",
    "```\n",
    "\n",
    "```\n",
    "ic| i: 6\n",
    "ic| half(a): 3.0\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All in all, if you find yourself using `print` to debug, you might find a one-time import of **icecream** followed by use of `ic` instead both more convenient and more effective."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **rich** for beautiful debugging\n",
    "\n",
    "[**rich**](https://github.com/willmcgugan/rich) is much more than just a tool for debugging, it's a way to create beautiful renderings of all objects both in the terminal and in interactive Python windows. You can use it to build fantastic looking command line interfaces. Here, we'll see how it can help us debug what value a variable takes *and* what methods can be used on a variable via its `inspect` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rich import inspect\n",
    "\n",
    "my_list = [\"foo\", \"bar\"]\n",
    "inspect(my_list, methods=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check out all of the many options for `inspect` using `help(inspect)`. We ran it here with `methods=True`, but there are plenty of other options.\n",
    "\n",
    "```{admonition} Exercise\n",
    "Create a dictionary (it doesn't matter what's in it, but you could map the integer 1 to the letter \"a\"). Then use `inspect` with `methods=True` to find out about all the methods you can call on an object of type dict.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Debugging with the IDE\n",
    "\n",
    "In this section, we'll learn about how your Integrated Development Environment, or IDE, can aid you with debugging. While we'll talk through the use of Visual Studio Code, which is free, directly supports Python, R, and other languages, and is especially rich, many of the features will be present in other IDEs too and the ideas are somewhat general. \n",
    "\n",
    "To begin debugging using Visual Studio Code, get a script ready, for example `script.py`, that you'd like to debug. If your script has an error in, a debug run will automatically run into it and stop on the error; alternatively you can click to the left of the line number in your script to create a *breakpoint* that your code will stop at anyway when in debug mode.\n",
    "\n",
    "To begin a debug session, click on the play button partially covered by a bug that's on the left hand ribbon of the VS Code window. It will bring up a menu. Click 'Run and debug' and select 'Python file'. The debugger will now start running the script you had open. When it reaches and error or a breakpoint it will stop. \n",
    "\n",
    "Why is this useful? Once the code stops, you can hover over any variables and see what's 'inside' them, which is useful for working out what's going on. Remember, in the examples above, we only saw variables that we asked for. Using the debugger, we can hover over any variable we're interested in without having to decide ahead of time! We can also see other useful bits of info such as the *call stack* of functions that have been called, what local (within the current scope) and global (available everywhere) variables have been defined, and we can nominate variables to watch too.\n",
    "\n",
    "Perhaps you now want to progress the code on from a breakpoint; you can do this too. You'll see that a menu has appeared with stop, restart, play, and other buttons on it. To skip over the next line of code, use the curved arrow over the dot. To dig in to the next line of code, for example if it's a function, use the arrow pointing toward a dot. To carry on running the code, use the play button.\n",
    "\n",
    "This is only really scratching the surface of what you can do with IDE based debugging, but even that surface layer provides lots of really useful tools for finding out what's going on when your code executes.\n",
    "\n",
    "You can find a short 'hello world!' debugging tutorial in the [official VS Code documentation](https://code.visualstudio.com/docs/python/python-tutorial#_configure-and-run-the-debugger)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logging\n",
    "\n",
    "Logging is a means of tracking events that happen when software runs. An event is described by a descriptive message that can optionally contain data about variables that are defined as the code is executing.\n",
    "\n",
    "Logging has two main purposes: to record events of interest, such as an error, and to act as an auditable account of what happened after the fact.\n",
    "\n",
    "Although Python has a built-in logger, we will see an example of logging using [**loguru**](https://github.com/Delgan/loguru), a package that makes logging a little easier and has some nice settings.\n",
    "\n",
    "Let's see how to log a debug message:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from loguru import logger\n",
    "\n",
    "logger.debug(\"Simple logging!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default message includes the time, the type of log entry message it is, what bit of code it happened in (including a line number), and the message itself (basically all the info we need). There are different levels of code messages. They are:\n",
    "\n",
    "- CRITICAL\n",
    "- ERROR\n",
    "- WARNING\n",
    "- SUCCESS\n",
    "- INFO\n",
    "- DEBUG\n",
    "- TRACE\n",
    "\n",
    "You can find advice on what level to use for what message [here](https://reflectoring.io/logging-levels/), but it will depend a bit on what you're using your logs for.\n",
    "\n",
    "What we've just seen are logging messages written out to the console, which doesn't persist. This is clearly no good for auditing what happened long after the fact (and it may not be that good for debugging either) so we also need a way to write a log to a file. This snippet of code\n",
    "\n",
    "```python\n",
    "logger.add(\"file_{time}.log\")\n",
    "```\n",
    "\n",
    "tells **loguru** to send your logging messages to a *log file* instead of to the console. This is really handy for auditing what happened when your code executed long after it ran. You can choose any name for your log file, using \"{time}\" as part of the string is a shorthand that tells **loguru** to use the current datetime to name the file.\n",
    "\n",
    "Log files can become quite numerous and quite large, which you might not want. Those logs from 6 months ago may just be taking up space and not be all that useful, for example. So, what you can also do, is tell **loguru** to use a new log file. Some examples of this would be `logger.add(\"file_1.log\", rotation=\"500 MB\")` to clean up a file after it reaches 500 MB in size, `rotation=\"12:00\"` to refresh the log file at lunch time, and `retention=\"10 days\"` to keep the file for 10 days.\n",
    "\n",
    "One further feature that is worth being aware of is the capability to trace what caused errors, including the trace back through functions and modules, and report them in the log. Of course, you can debug these using the console, but sometimes having such complex errors written to a file (in full) can be handy. This example of a full traceback comes from the **loguru** documentation. The script would have:\n",
    "\n",
    "```python\n",
    "logger.add(\"out.log\", backtrace=True, diagnose=True)  # Caution, may leak sensitive data if used in production\n",
    "\n",
    "def func(a, b):\n",
    "    return a / b\n",
    "\n",
    "def nested(c):\n",
    "    try:\n",
    "        func(5, c)\n",
    "    except ZeroDivisionError:\n",
    "        logger.exception(\"What?!\")\n",
    "\n",
    "nested(0)\n",
    "```\n",
    "\n",
    "while the log file would record:\n",
    "\n",
    "```\n",
    "2018-07-17 01:38:43.975 | ERROR    | __main__:nested:10 - What?!\n",
    "Traceback (most recent call last):\n",
    "\n",
    "  File \"test.py\", line 12, in <module>\n",
    "    nested(0)\n",
    "    └ <function nested at 0x7f5c755322f0>\n",
    "\n",
    "> File \"test.py\", line 8, in nested\n",
    "    func(5, c)\n",
    "    │       └ 0\n",
    "    └ <function func at 0x7f5c79fc2e18>\n",
    "\n",
    "  File \"test.py\", line 4, in func\n",
    "    return a / b\n",
    "           │   └ 0\n",
    "           └ 5\n",
    "\n",
    "ZeroDivisionError: division by zero\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "interpreter": {
   "hash": "e82e9306cec5f3813db66e68c531c869f324183486c55e9589deb7f101067873"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('codeforecon': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
